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1. Introduction 


Data from Montana’s 2011 Traffic Safety Problem Identification Report (Montana Department of 
Transportation 2011) suggested that seat restraint usage increased steadily prior to 2002. Since then, 
however, that trend has stagnated, with seat belt use rates remaining around 78-81 percent. Seat belt 
compliance rates decreased slightly from 79.2 to 78.9 percent in 2010, and Montana recorded 22 more 
fatal crashes statewide in 2011 than in 2010, according to Montana’s Highway Traffic Safety 2011 
Annual Report. It has been speculated that the increased crash fatality events may be associated with 
declining seat belt use. 


Montana is a low seatbelt use state. Nonetheless Montana’s seat belt usage is relatively high for a state 
with a secondary seat belt law, which allows law enforcement to ticket for failure to buckle up only when 
stopping a motorist for another reason. According to the Montana Highway and Traffic Safety 2011 
Annual Report, media attention surrounding proposed legislation to upgrade Montana’s occupant 
protection laws brought attention to the secondary enforcement provisions in the current laws. This may 
be a contributing factor in the stagnant or decreasing use of seat belts across the state. 


According to the 2011 Annual Report, Montana recognizes that substantial progress has not been made in 
increasing seat belt usage. In September of 2011, a Special Management Review conducted by the 
National Highway Traffic Safety Administration (NHTSA) looked at Montana’s occupant protection 
programs. From this review came a suite of recommendations focused on improving Montana’s seat belt 
usage rates that includes: efficient administration; and effective planning, programming, implementation 
and evaluation of activities involving saving lives. The Montana Department of Transportation (MDT)’s 
analysis of the state’s fatal and incapacitating injury data provided a clear vision of Montana’s major 
traffic safety challenges: impaired driving and low seat belt use. 


The MDT State Highway Traffic Safety Section is focused on improving seat belt use and has set forth 
the following occupant protection goals: 


e Reduce the five-year average number of unrestrained vehicle occupant fatalities from 126 in 2010 
to 98 by 2015. 


e Reduce the five-year average number of unrestrained vehicle occupant fatalities and 
incapacitating injuries from 633 in 2010 to 490 by 2015. 


e Increase the annual statewide seat belt use for front seat passenger vehicle occupants from 78.9% 
in 2010 to 89.3% by 2015. 


To achieve these goals, MDT needs to allocate resources to the most impactful programs, yet the state did 
not have a formal occupant protection program evaluation process in place. The objective of this project 
was to provide MDT with a quantitative assessment of the impact each occupant protection program has 
on seat belt use in Montana. 
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2. Literature Review 


Introduction 


There is widespread agreement that vehicle occupant protection is a critical component of public health, 
and seat restraint use is broadly accepted as a major contributor to occupant protection (Houston & 
Richardson, 2005). Despite the widely accepted efficacy of seat restraint usage in preventing automobile- 
related mortalities, nationwide occupant protection rates have stagnated in the last 15 years. Current data 
show that for adult drivers and passengers, seat belt use nationwide was 85 percent in 2010 (NHTSA, 
2011), with compliance exceeding 90 percent in 17 states and below 70 percent in New Hampshire, 
Wyoming, and American Samoa (NHTSA, 2011). This variation across states is often attributed to 
differences in state-specific laws (i.e., primary vs. secondary enforcement laws), as well as a variety of 
state-specific programs intended to improve seat belt compliance. 


State and federal transportation agencies have confronted the stagnation in seat belt compliance rates with 
a suite of programs designed to increase their use (NHTSA, 2001; Preusser Research Group, 2002). The 
effectiveness of these programs remains unclear. In order to efficiently allocate financial and time 
resources to the most effective programs, and avoid wasting resources on programs that have little impact, 
state-level seat restraint compliance programs need to be subjected to rigorous and thoughtful evaluation. 
The intent of this document, which is specifically geared toward evaluating occupant protection protocols 
in the state of Montana, is to (1) define and motivate program evaluation; (2) define a set of approaches 
that could be used to evaluate a program; (3) review the essential elements of strong program evaluation; 
(4) provide guidance on evaluation methodologies; and 5) provide two detailed examples of effective 
program evaluation protocols. 


The Montana Department of Transportation (MDT) has funded a variety of occupant protection safety 
programs, mainly geared toward improving seat belt usage rates. To this end, the MDT State Highway 
Traffic Safety Section has set forth the following occupant protection goals: 


e Reduce the five-year average number of unrestrained vehicle occupant fatalities from 126 in 2010 
to 98 by 2015. 


e Reduce the five-year average number of unrestrained vehicle occupant fatalities and 
incapacitating injuries from 633 in 2010 to 490 by 2015. 


e Increase the annual statewide seat belt use for front seat passenger vehicle occupants from 78.9% 
in 2010 to 89.3% by 2015. 


Although the State Highway Traffic Safety Section has implemented many programs intended to help 
meet these goals, formal evaluation of these programs’ performances has not been prioritized until now. 
In this document, our intention is to provide a specific framework for program evaluation that is 
consistent with the goals and resources available to the MDT State Highway Traffic Safety Section. 
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Specifically, this document provides a literature review summary of journal articles and technical reports 
to understand what evaluation means in this context, match each program to an appropriate evaluation 
technique, provide generic step-by-step instructions for program evaluation of occupant protection 
programs, and give some examples of well-designed and well-executed program evaluations from 
Montana and other states. 


Background 
Seat Belts for Adults - Trends and Laws 


All new passenger cars in the United States offered some form of seat belts beginning in 1964, shoulder 
belts in 1968, and integrated lap and shoulder belts in 1974 (ACTS, 2001). Few occupants wore the belts; 
surveys in various locations recorded belt use of about 10 percent. The first widespread survey, taken in 
19 cities in 1982, observed 11 percent seat belt use for drivers and front-seat passengers (Williams & 
Wells, 2004). This survey became the benchmark for tracking belt use nationwide. 


New York enacted the first belt use law in 1984. Other states soon followed. In a typical state, belt use 
rose quickly to about 50 percent shortly after the state’s belt law went into effect. However, during the 
year following the effective date of the law, the seat belt use rate usually decreased slightly, by about four 
percentage points on average (Nichols, 2002). 


High-visibility, short-duration belt law enforcement programs, often called “STEPs” (Selective Traffic 
Enforcement Programs), “STEP waves,” or “blitzes,” were demonstrated in individual communities in the 
late 1980s. North Carolina’s Click It or Ticket program took this model statewide beginning in 1993 and 
raised the use rate above 80 percent (Williams & Wells, 2004). Statewide, multi-state, and national 
enforcement programs increased through the 1990s under different names and sponsors. These 
enforcement programs typically raised belt use by 13 to 26 percentage points, with greater gains where 
belt use was lower (Dinh-Zarr et al., 2001; Nichols, 2002). Belt use often decreased by about six 
percentage points after the enforcement program ended. 


The Click It or Ticket model expanded nationwide in 2003 (Solomon et al., 2004). Programs operating 
under this model have used communications and outreach techniques such as extensive paid advertising, 
and have included strategies designed specifically to increase seat belt use among low-belt-use groups 
such as pickup truck drivers (Nichols et al., 2009), teens, and rural residents (Nichols et al., 2009). The 
national belt use rate reached 84 percent in 2009 (NHTSA, 2011). More recent research found that belt 
use, measured as observed or self-reported belt use or belt use in fatalities, increased nationwide and in 
almost all states during the 2000-2006 time period when these mobilizations were in operation (Tison & 
Williams, 2010). Importantly, Tison and Williams (2010) also concluded that the Click It or Ticket 
mobilizations conducted during these years were an important factor in these increases in seat belt use. 


Recent research has focused on the contrasts between daytime and nighttime crashes in terms of fatality 
rates and restraint use. According to 2012 Fatality Analysis Reporting System (FARS) data, almost two- 
thirds (61 percent) of people killed at nighttime did not use restraints (NHTSA, 2013a). In contrast, the 
percentage of fatally injured passenger vehicle occupants during daytime crashes who were unrestrained 
was just under half (47 percent) (Varghese & Shankar, 2007). Furthermore, the FARS data indicated that 
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for the 10-year period from 1998 to 2007, nighttime seat belt use was on average 18 percentage points 
lower than daytime belt use (Tison et al., 2010). Strategies to increase restraint use among nighttime 
drivers are currently being developed, implemented, and evaluated. 


Table 2.1 provides a breakdown of common occupant protection programs targeted at adults broken down 
by type of countermeasure. The effectiveness is measured by the reductions in crashes or injuries (unless 
noted otherwise). The use column shows the popularity of the type of countermeasure with low, medium 
and high levels among all states. The cost column shows the cost of implementation with levels low, 
medium and high, from essentially no cost (maintain the same facility, faculty, etc.) to high costs of hiring 
new staff and purchasing equipment. Finally, the time column shows the implementation time period, 
excluding legislation and policy time commitments. 


Table 2.1. Countermeasures for Adults (NHTSA, 2013b). 
1. Seat Belt Use Laws 


Countermeasure | Effectiveness | Use__—| Cost__—'|[ Time __ 


1.1 State primary enforcement belt use laws | eke [Medium | Low [Short | 
1.2 Local primary enforcement belt use laws 
1.3 Increased belt use law penalties 
1.4 Coverage: seating position, vehicles, ages fe | Medium | tow [Short | 


2. Seat Belt Law Enforcement 


Countermeasure | Effectiveness | Use — | Cost —s | Time. 
2.1 Short high-visibility belt law enforcement | %* * KK 


2.2 Combined enforcement, nighttime KKKK | Unknown | High | Medium _| 


Used in many jurisdictions but often only once or twice each year 


3. Communications and Outreach 


Countermeasure | Effectiveness  |Use  _—| Cost | Time _— 
3.1 Supporting enforcement KKKKK | Medium | Varies | Medium | 


3.2 Strategies for low-belt-use groups KKK KT | Unknown | Unknown | Medium _| 


t ¥& © For stand-alone programs not supporting enforcement 


4. Other Strategies 


Countermeasure [Use [Cost [Time | 
1 eetve poor KKK KT [tow [Vanes [wes 
1 ZEmployer rors AIK 


In low-belt-use settings with no belt use law 
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As of August 2010, all states except New Hampshire require adult passenger vehicle occupants to wear 
seat belts. The laws in 31 states plus the District of Columbia permit law enforcement officers to stop 
vehicles based solely on observed belt law violations; these are called “standard” or “primary” 
enforcement laws. The remaining 18 states have secondary enforcement laws that allow nonusers to be 
cited only after they first have been stopped for some other traffic violation (Insurance Institute for 
Highway Safety, 2010). In 2010, minimum fines in primary law states ranged from $10 to $200 with a 
fine of $25 or more in all but eight states. Minimum fines in secondary law states ranged from $10 to $75 
with a fine of $25 or less in all but four states (Insurance Institute for Highway Safety, 2010). 


What is Evaluation and Why: Creating the Paradigm 


Evaluation is a formal framework for designing a program, critiquing its performance, and providing 
suggestions on how it could be improved. A good evaluation rests on regularly reviewing a series of steps 
over the course of a program’s design and implementation. Planning evaluation strategies early in a 
program’s conception is an important first step, not only toward evaluating the program, but toward 
designing a strong program from the start. Evaluations are essential, since they inform the program’s 
designer and advocate, and also its funders, about whether the program works. By evaluating programs 
early and often, implementers can identify small problems before they get out of hand. 


Evaluation does not need to be a massive undertaking. As detailed below, in some cases administrative 
evaluations are sufficient to measure whether a program’s goals are met. Also, if goals are clearly 
articulated at a program’s outset, then collecting the necessary data can be a simple task with limited time 
commitment. Finally, evaluations provide strong measures of program efficacy for programs that work. 
As such, they are an important tool for justifying program continuation and expansion to funders. 
Evaluation is the simplest way to clearly show the public that a program works. Figure 2.1 provides steps 
for evaluating program effectiveness (NHTSA, 2008). 


Steps for evaluating program effectiveness 

1. Accurately identify the problem 

2. Uncover some problems you didn't know you had 

3. Establish reasonable, practical objectives for dealing with these problems 
4. Determine if you accomplished your program objectives 


5. Provide information to funding sources, the media and the public to continue 
support for program 


6. Determine if and how a program should be revised to increase effectiveness. 


-- from The Art of Appropriate Evaluation: A Guide for Highway Safety Program 
Managers (DOT HS 811 061) 


Figure 2.1. Steps for Evaluating Program Effectiveness 
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Finding the Appropriate Evaluation Technique 


When planning an evaluation, it is important to consider exactly what objectives are being evaluated. 
There are three levels on which occupant protection program evaluations can focus. At the most basic 
level, programs can be assessed on whether they were implemented as planned. At a slightly higher level, 
programs can be evaluated in terms of whether they were effective at actually altering compliance of an 
accepted safety measure. At the most esoteric level, programs can be assessed in terms of whether they 
saved lives (see Figure 2.2). 


Levels of Evaluation Complexity 


Evaluation of program administration 


Question addressed: Were resources used as intended? 


Evaluation of whether compliance was 
increased 


Question addressed: Did observed compliance rates increase? 


Evaluation of whether lives were saved 


Question addressed: Did the intervention decrease vehicle- 
induced mortality rate?? 
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Figure 2.2. Levels of Evaluation Complexity 
Administrative Evaluations 


An administrative evaluation tracks whether a program was implemented as planned. For example, 
suppose you intended to conduct several media blitzes to promote occupant protection in an area. An 
administrative evaluation of that program would include the number of media hours purchased and media 
units presented, and perhaps a report of how many listeners, readers, or watchers the blitz was expected to 
reach. Among the MDT occupant protection programs, media expenditures might be particularly 
amenable to administrative evaluation. 


Another important element of program evaluation is budget: were resources spent in a manner consistent 
with the project’s proposed budget? Whether a project adheres to its proposed budget provides an 
indication of whether it is being implemented as intended. 


Evaluations of Compliance Improvement 


The challenge in evaluating whether an occupant protection program actually reduced traffic deaths is the 
presence of numerous confounding variables, paired with the small number of fatalities that occur in 
communities of small-to-moderate size. While the presence of confounders can be circumvented, doing so 
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often requires substantial complexity and cost in study design. Therefore, evaluating the impact of an 
intervention on actual traffic deaths (for example) should be done only in those cases where the 
intervention is not already broadly known to impact the outcome, or for interventions that haven’t been 
tried elsewhere. 


Most accepted occupant protection methodologies have been subject to national studies that clearly 
illustrate they can save lives. For example, it is not necessary to show that seat belt use saves lives, since 
that is already documented. However, it may be necessary to show that a particular program increases seat 
belt use compliance (i.e., did you accomplish your objectives) (NHTSA, 2008), and at a more basic level, 
that resources allocated to promoting seat belt use did actually get used to promote seat belt usage (i.e., 
did you implement the program as planned) (NHTSA, 2008). Figure 2.3 contains a list of interventions 
already shown to be effective. For these interventions, it is sufficient to simply show that a program 
improves compliance, as opposed to showing that a program saves lives. 


Interventions already shown to save lives 


If a program involves improving compliance in one of these interventions, 
showing improvement in compliance is sufficient for showing program success 


- Safety belts - Sobriety checkpoints 
- Child safety seats - Tougher impaired driving laws 


- Bicycle helmets - Crossing guards 


- Motorcycle helmets - Traffic calming devices 
(e.g., speed bumps) 
- DWI enforcement - Educating judges and prosecutors 


-from The Art of Appropriate Evaluation: A Guide for Highway Safety Program 
Managers 
DOT HS 811 061. 


Figure 2.3. Effective interventions. 


Detailed examples of cases where programs were evaluated in terms of the rate at which they improved 
seat belt compliance are included in the Case Studies section. 


Evaluations of Whether Lives were Saved 


Evaluations that directly link an intervention to traffic mortality rates can follow one of several 
approaches. One approach focuses on the physics of the implementation. Under this method, researchers 
ask should the device or intervention save lives? This approach can be tested with crash dummies in a 
laboratory setting or via computer simulations of a crash scenario. For example, Park et al. (2010) 
conducted a detailed autopsy and a follow-up computer simulation study after an unrestrained subject 
died of a neck fracture in a traffic crash. Findings from evaluations like this can lead to device-specific 
refinements or regulations. In the Park et al. (2010) study, for instance, the evaluators found that in 
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settings where subjects’ heads hit the roof of the car, lower speeds are required to induce neck fractures 
than in cases where subjects’ heads do not hit the roof. 


A second approach relies on observational studies of fatal crash reports. For example, House et al. (2012) 
recently examined the effectiveness of booster seats in decreasing motor vehicle collision-related fatalities 
in children four to eight years of age. To do so, they speculated that if booster seats actually reduced child 
injury rates, then children in booster seats should visit the emergency room less frequently following 
motor vehicle collisions than children who were not similarly restrained. This study found no relationship 
between use of booster seats and decreased ER visit rate. This finding corroborated results from Rice et 
al. (2009). In that study, researchers used a matched cohort design, relying on FARS data for mortalities, 
and comparing outcomes for children in booster seats vs. not in booster seats within the same vehicle. 
This structure is possible using large-scale datasets like FARS, which contain sufficient data to detect 
trends, but is infeasible for smaller-scale programs where only a very limited number of fatalities occur. 


Two Guiding Principles 


Two common threads emerge from the two more complex evaluation approaches described above. These 
are: 


1. Set measurable objectives. 
One of the major challenges associated with determining and designing an appropriate evaluation 
is identifying a project’s goals. Goals tend to be written in language designed to persuade funders. 
For example, a common goal in occupant protection programs might be to “reduce traffic deaths.” 
Unfortunately, goals expressed in such grandiose terminology tend to be difficult to evaluate, 
because a limited number of fatalities tend to occur. A stronger measurable objective, for 
example, would be an increase in observed seat restraint usage upon completion of program 
implementation. 

2. Collect data prior to program implementation. 
In general, an evaluation should compare compliance before and after the implementation was 
enacted. Collecting baseline data provides a reference group and allows statements like, “Before 
implementation, seat belt compliance was [X%J]; following program implementation, compliance 
rose to [Y%], so we believe the program improved compliance by [Y-X%J].” This allows one to 
clearly state whether the program has accomplished its goals. 


By considering measurable project objectives and collecting preliminary data, program managers can 
dramatically improve their programs’ potential for effective evaluation. 


A Preliminary Strategy for Evaluating the Montana Programs 


In this project, our aim is to evaluate the effectiveness of various MDT occupant protection programs. 
Table 2.2 lists the different MDT occupant protection compliance programs and our likely approach to 
evaluating their efficacy. All programs are amenable to administrative evaluations; however some 
programs may also be evaluated in terms of whether they actually improved seat belt compliance. This is 
a preliminary list that is subject to change, depending on the data available for each program. 
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Table 2.2. MDT's Occupant Protection Programs and Evaluation Methods 


MT-DOT Affiliated Seatbelt Compliance Programs and 
Evaluation Methods 


Buckle-Up Montana Administrative/Compliance 


Selective Traffic Enforcement Program (STEP) Administrative/Compliance 


Earned Media Administrative/Compliance 
Paid Media Administrative/Compliance 
Safe on all Roads (SOAR) (insufficient available data) 


Office of Public Instruction's (OPI) Driver Education | Administrative/Compliance 
Curriculum Update 


Step-by-Step Guide to Program Evaluation 


The steps to effective program evaluation are outlined in Figure 2.4. This section will describe each step 
as it pertains to occupant protection programs. 


Steps to Effective Program Evaluation 
. Identify the problem 4. Gather baseline data 


. Develop reasonable objectives 5. Gather post-program 
data and analyze results 


. Develop a plan for measuring results 6. Report Results 


-from The Art of Appropriate Evaluation: A Guide for Highway Safety Program 
Managers 
DOT HS 811 061. 


Figure 2.4. Steps to Effective Program Evaluation 


1. Identify the problem 


Take time to understand the underlying issues at hand—this is a critical step in creating measures for 
future prevention. Proper problem identification will allow focusing on details of the program 
including magnitude of the problem, target audience, appropriate countermeasures and developing an 
understanding of the baseline data that ought to be collected. 


2. Develop reasonable objectives 


All occupant protection programs have the same goal: save lives. This is obviously a very important 
objective, but it is very difficult to measure whether a program met this goal due to the variety of 
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other factors that could influence vehicle fatalities during a given time period. Therefore, specific, 
measurable goals should be created, such as improving seat belt compliance. In addition, these goals 
should be subject to a deadline that is in line with the program objectives. 


3. Develop a plan for measuring results 


From your reasonable (and measurable) objectives, a plan for measuring results will follow easily. 
For example, if the goal is to improve seat belt compliance, then measuring the results should consist 
of collecting data on vehicles with occupants using/not using seat belts. At this point it is also 
important to determine how the data will be collected and how the analysis will be performed. 


4. Gather baseline data 


To measure the effectiveness of a program, it is necessary to acquire baseline data to which post- 
program results can be compared. It may also be of interest to collect data on other measures, such as 
through a public opinion survey, to understand changes that result from your program. 


5. Gather post-program data and analyze results 


During this stage of the evaluation it is important to ensure data collection is going as planned and 
that enumerators are collecting the appropriate data on the correct terms. 
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6. Report Results 


Congratulations—this is the final leg of the journey! Carefully consider the audience that results will 
be presented to and what message is to be conveyed. 


Case Studies 


Nevada 


The state of Nevada, which like Montana has a secondary seat belt law, was ranked third among all states 
(including states with primary seat belt laws) in 2005 for its observed seat belt use, and second among 
states with secondary seat belt laws (Subramanian, 2005; Vasudevan & Nambisan, 2005). This 
achievement can be attributed to a combination of education, outreach and enforcement campaigns 
(Vasudevan & Nambisan, 2005; Houston & Richardson, 2006), and demonstrates that even states with a 
secondary law can have very high seat belt use. 


1. Identify the problem 


Nevada wanted to improve its seat belt compliance rates without moving to a primary seat belt law. State 
officials were particularly interested in the efficacy of media and enforcement campaigns in increasing 
seat belt compliance. 


2. Develop reasonable objectives 


The state’s primary aim was to increase seat belt compliance through media and enforcement campaigns. 
It designed surveys that estimated changes in seat belt compliance with a 5 percent error margin. 


3. Develop a plan for measuring results 


The state of Nevada’s Office of Traffic Safety studied the efficacy of media and enforcement pulses by 
initiating pre- and post-campaign surveys. It surveyed a set of sites each year, and then initiated its media 
and enforcement campaign (which was synchronous with NHTSA’s). After the enforcement campaign, it 
conducted a post-campaign set of surveys at the same sites and with the same sampling intensity as the 
pre-campaign surveys. 


4. Gather baseline data 


In Nevada, evaluators had access to data on about 40,000 vehicles between 2003 and 2005 (Vasudevan, 
2009). The Office of Traffic Safety has collected pre- and post-campaign surveys since 2002. Starting in 
2003, it followed the Click It or Ticket guidelines established through NHTSA. Surveys were balanced, 
so that all observation sites were observed both pre- and post-campaign, and the same number of 
observations was collected at each site for each set of surveys (Vasudevan, 2009). It conducted a power 
analysis to show what level of sampling intensity would be necessary to draw given inferences. It 
determined that a sample size of 385 observations per site would be sufficient to detect 50 percent seat 
belt compliance with a 5 percent error rate at the 95 percent level (Vasudevan, 2009). This power analysis 
allowed researchers to designate exactly how much sampling effort was necessary to evaluate the 
programs’ effectiveness with a given level of statistical confidence. 
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5. Gather data and analyze results 


To compare the pre- and post-survey seat belt compliance data, Vasudevan (2009) used a binomial test 
for the difference in two proportions. 


6. Report results 


The binomial test conducted by Vasudevan (2009) provided convincing evidence that seat belt 
compliance was much higher during the post-campaign surveys than in the pre-campaign surveys 
(Vasudevan, 2009 Table 2.3). This was true in every year studied, and suggests that the campaigns were 
effective at increasing seat belt compliance. Results were reported to the state, and eventually were 
released to the broader community through a peer-reviewed publication (Vasudevan, 2009). 


Chemung County, New York: 90 Percent Compliance in Three Weeks 


Despite years of public information, education and enforcement campaigns, seat belt use rates in the 
United States remained at low levels in the late 1990s. The rate in Chemung County, New York, was 63 
percent in 1999. 


1. Identify the problem 


Chemung County wanted to raise its seat restraint compliance rate. 


2. Develop reasonable objectives 


In October 1999, , and the Insurance Institute for Highway Safety conducted a STEP called “Buckle Up 
NOW” in Chemung County, New York, with the objective of demonstrating that seat belt use can be 
increased to 90 percent or more in a period of three weeks. 


3. Develop a plan for measuring results 


A plan was developed to collect data on observed seat belt use before the campaign and daily for the three 
weeks during the campaign. Also, data was to be collected on enforcement activity, paid and earned 
media, self-reported seat belt use, attitudes related to the seat belt law, and the public’s recollection of 
activities undertaken to encourage compliance with the law. 


4. Gather baseline data 


Before the program began, a seat belt observation form was used to estimate the belt use rate on county 
roads. While the program was underway, observers stood at the same street corners at the same time each 
day to record belt use of the first 50 passing vehicles. 


5. Gather data and analyze results 


Post-program data was collected and analyzed. This included seat belt observational surveys conducted 
before, during and after the enforcement campaign, reports by participating enforcement agencies on the 
amount and type of enforcement carried out, tabulations of the amount and type of media purchased for 
the campaign as well as the amount and type of earned media generated. Also, public awareness surveys 
assessed who noticed the enforcement and media efforts, and how much support existed for seat belt laws. 
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6. Report results 


Several articles and reports were written to publicize the results to the general public, funders and other 
agencies that had interest in transportation safety. Seat belt use rates during the program rose from 63 
percent at baseline to 75 percent after the first week of enforcement, to 84 percent after the second week, 
and to 90 percent after the third week. In addition, opinion surveys found that 90 percent of respondents 
were aware of the safety belt program. Public perception that the belt law was being enforced “very 
strictly” increased from 34 percent before “Buckle Up NOW!” to 77 percent after the program, and 79 
percent supported enforcement to increase seat belt use. 


Literature Gaps and Potential Extensions 


This literature review revealed two issues that current program assessment protocols do not treat in depth, 
but which are nonetheless relevant when assessing traffic safety programs in general. First, we found little 
precedence on protocols for compiling program and surveillance data across the spatial and temporal 
domains of interest. Second, the statistical methodology employed in currently published analyses 
typically does not account for a single community hosting multiple programs geared toward achieving the 
same objective. When assessing the efficacy of a single program, it is important to use methods that 
differentiate between all the programs present in a single community. While statistical techniques for 
estimating program-specific effects, and identifying treatment complexes with particular synergy, are 
regularly used in other fields (Gelman & Hill, 2007), we found limited application of these techniques 
within the transportation safety domain (e.g., Pulugurtha & Repaka, 2008). In this section, we discuss 
each of these issues, with an emphasis on how we anticipate accounting for these gaps in the assessment 
of MDT's occupant protection (“OP”) programs. 


Compiling Data in a GIS 


In order to account for a suite of different programs operating on the same site at the same time, we must 
first establish when and where different OP programs operated. In the case of Montana's OP programs, 
we specifically need to link a set of seat belt survey sites to the presence of local OP programs. This is not 
a trivial task: while seat belt survey sites are explicit points in space, occupant protection programs 
operate across a spatial (and a temporal) range. For example, one set of seat belt survey data was collected 
at the intersection of Peach Street and Wilson Avenue (survey site C-44) in Bozeman. Seat belt 
compliance at this site may likely be influenced by programs specific to Bozeman, as well as programs 
operating throughout Gallatin County. Another seat belt survey data set, collected at site P-14 on 
Highway 191, would be influenced by the same Gallatin County programs as the Bozeman site, but may 
not be related to the Bozeman-specific programs. Geographic Information Systems (GIS) provide an 
efficient means of linking programs operating at different spatial scales to surveys that occurred at very 
specific points. GIS are composed of a set of layers, each of which depicts a particular data stream. 
Layers can be visualized as maps, showing points, lines, or polygons coded to depict particular levels of a 
given attribute (for example, a polygon layer of Montana high school districts could be coded so that each 
school district's color indicated its Drivers Education completion rate). 
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In the MDT OP program case, one GIS layer consists of the seat belt survey points, with additional 
separate layers for each OP program under assessment (for example, a layer indicating regions covered by 
Buckle Up Montana coalitions, a layer indicating county-level STEP expenditures, etc.). After building 
separate layers for each program being assessed, statistical software can be used to “drill down” through 
the GIS and extract the values associated with every individual layer for each survey site. Drilling down 
gives us a unique set of local program measures (which could be presence/absence or intensity, depending 
on available data), linked to each seat belt surveillance site. 


Assessing the Role of Multiple Programs Simultaneously 


Extensive statistical methodology exists for separately estimating the role of a suite of treatments (in the 
MDT case, different occupant protection programs) simultaneously. While some of these methods are 
slightly more sophisticated than the binomial tests often used in published program assessments, we 
found the application of these techniques to be quite limited. In general, the approach we plan to take is 
that of a generalized linear model (GLM), in which the seat belt use rate at a given site acts as the 
response variable, and a separate linear predictor is estimated for each occupant protection program. In 
the next paragraphs, we explain how GLMs allow for estimating the role of specific effects, even when a 
suite of programs operates concurrently in space and time. 


In the context of seat belt survey data, a binomial test compares seat belt compliance in a sample from a 
population that was not subjected to any occupant protection programs (untreated), to compliance in a 
sample from a population where a treatment (i.e., OP program) was applied. This test assumes that the 
only difference between the two populations sampled was that one received the treatment (i.e., had an 
occupant protection program operating locally), and the other did not. Essentially, this test compares 
mean compliance in the treated population to mean compliance in the untreated one. However, in settings 
where more than a single treatment differentiates sites, this model is insufficient. In statistical 
methodology, a GLM is a direct extension of a binomial test that accounts for multiple predictors (i.e., 
multiple programs) in the context of a single model. 


For the GLM case, compliance at a given survey site is assumed to be a function of the particular 
combination of OP programs present in its locality. In particular, a GLM can treat the seat belt 
compliance rate at a given site as a function of the effect of OP program 1 at that site, plus the effect of 
OP program 2 at that site, plus the effect of OP program 3 at that site, etc. The GLM assigns a unique 
model parameter to each program, and seat belt survey and program data are used to estimate the 
parameter associated with each program separately. After model fitting, programs with parameters 
estimated to be indistinguishable from 0 are inferred to not influence seat belt survey outcomes. The OP 
programs associated with positive parameter values are interpreted as increasing compliance, and 
programs associated with negative parameters are interpreted as decreasing compliance. This modeling 
framework also allows for confidence interval estimation associated with each parameter, which provides 
a means of assessing the statistical significance of all estimated effect. 


In the GLM structure described above, the various OP programs are assumed to impact seat belt 
compliance independently (i.e., OP program 1 has the same effect on compliance regardless of whether or 
not OP program 2 is present locally). By incorporating program-by-program interaction terms into the 
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GLM, we can estimate whether two particular programs actually operate synergistically or 
antagonistically when present in the same locality. Employing a GLM structure, as opposed to a binomial 
test, allows us to make much more refined inferences about the role of the various OP programs in 
actually altering seat belt compliance. Under this model structure, we attain a much broader scope of 
inference (i.e., we can make far more specific statements about program efficacy) with relatively small 
costs to model complexity. 


Conclusions 


The intent of this chapter was to define and motivate program evaluations, illustrate a few methods for 
evaluation with an emphasis on programs currently used to promote seat restraint usage by the MDT State 
Highway Traffic Safety Section, outline a protocol for strong program evaluation, and provide several 
detailed examples of program evaluations in the occupant protection context. The research team wishes to 
underscore the importance of any form of program evaluation. While future work on this project will take 
sophisticated approaches to program evaluation, the research team emphasizes that even the simplest 
program evaluation can provide critical information about program efficacy and implementation. In 
general, much can be gained through a simple and timely critical assessment of all programs, no matter 
their size or objective. 
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3. Quantitative Assessment of MDT’s Occupant Protection 
Programs 


Introduction 


States invest extensively in occupant protection programs, yet the impact these programs actually have on 
improving seat belt compliance rates remains unclear. This report describes an analysis of National 
Highway Traffic Safety Administration (NHTSA) seat belt survey data in relation to the presence and 
structure of local occupant protection programs utilized in Montana. The period of 2010 to 2012 was 
chosen because these were years for which extensive electronic records were available. Analysis and data 
aggregation were conducted in concert with ongoing input from MDT. Although seat belt usage was on 
the rise in Montana prior to 2002, it has stagnated at between 76 percent and 81 percent compliance since 
that time (Montana Department of Transportation, 2011). Similar trends have been observed at the 
national level. 


Small declines in compliance (from 79.2 percent to 78.9 percent), paired with increases in fatal crash 
incidence (22 additional crashes) from 2010 to 2011 brought seat restraint compliance concerns to the 
forefront of MDT priorities. A 2011 NHTSA assessment recommended that MDT enact a suite of 
measures aimed at improving compliance. Among these was an evaluation of existing occupant protection 
programs. 


Both NHTSA and MDT have expressed the need to increase compliance rates in Montana. In 2013, 
MDT’s stated goal was to increase annual observed compliance rates to 89.3 percent by 2015 (Figure 
3.1). 
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Figure 3.1. Percent of outboard, front seat passenger vehicle occupants using a seat belt by year. 
Source: Montana 2013 Traffic Safety Problem Identification Report. 


To complement the state’s secondary seat restraint law, MDT runs a mixture of occupant protection 
programs aimed at increasing seat restraint use. These programs each use a combination of education, 
outreach and enforcement to achieve their goals, all of which are key for increasing seat belt usage 
(Vasudevan & Nambisan, 2005; Houston & Richardson, 2006). The state commits resources to each of 
these strategies. In the education domain, the Montana Office of Public Instruction (OPI) provides 
voluntary driver education instruction throughout the state. In Montana, all high schools are eligible to 
obtain driver education certification and apply for driver education funding. However, only about 80 
percent of high school districts do this. State-sponsored media campaigns are enacted at regular intervals 
throughout the year. The Selective Traffic Enforcement Program (STEP), a program conducted by MDT, 
provides financial resources to applicant enforcement agencies to allow for additional targeted 
enforcement efforts at times and places that local enforcement personnel deem necessary. In this 
investigation, the task was to evaluate the efficacy of MDT's various occupant protection programs. 
Specifically, the consultants were asked to 1) estimate the effect each program had on local compliance 
individually, and 2) identify combinations of programs associated with particularly strong increases in 
local seat restraint compliance rates. 
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Data Preparation 


The objective of this analysis was to quantitatively compare seat belt usage in places and times where 
MDT programs were present to places and times where they were not present. This analysis hinges on the 
assumption that “effective” programs lead to increased seat belt usage in surrounding areas. To achieve 
this objective, the research team prepared data and constructed a statistical model that quantitatively 
compared seat belt usage at sites impacted by specific MDT programs to unimpacted sites. 


Data from multiple sources were used to conduct the quantitative analysis, and this section contains a 
detailed description of each dataset. Data from NHTSA’s National Occupant Protection Usage Survey 
(NOPUS) provided a measure of seat belt compliance rates at sites throughout Montana over multiple 
years. This analysis incorporated NOPUS seat belt compliance observations as dependent, or “response,” 
variables. As such, the NOPUS data provided a means of comparing seat belt compliance rates for sites 
influenced by different occupant protection programs. MDT provided information on each of its occupant 
protection programs, which the research team translated into quantitative measures of program effort as 
described below. Data from other sources (e.g., U.S. Census data; weather data from 
weatherunderground.com) were included to account for other factors likely associated with seat restraint 
use rates. 


NOPUS Data on Seat Belt Usage 


The National Highway Traffic Safety Administration initiated NOPUS in the mid-1980s, and continues to 
conduct NOPUS sampling on a semiannual basis. The NOPUS sampling events are conducted at sites 
across the United States. Survey events are controlled for time of day and day of week, and are conducted 
at the same sites every late-April/early-May and every June. In Montana, there are 120 NOPUS sites (12 
Interstate highway sites, 24 National Highway System (NHS) sites, 20 secondary/county sites and 64 city 
sites; see Figure 3.2). These sites are located in 30 of Montana’s 56 counties, and in 19 cities/towns 
throughout the state. Fifty-five of the 120 sampling sites are in regions the U.S. Census Bureau classifies 
as rural (U.S. Census Bureau, 2010). In this analysis, seat belt compliance rates from NOPUS were used 
to compare Montana’s occupant protection programs. For each NOPUS sampling event, the research team 
identified a set of demographic and occupant-protection-program-related factors operating in the event’s 
vicinity. Seat belt compliance rates for sites where programs were present were then compared to sites 
where programs were absent via a logistic regression model with random effect terms for NOPUS site and 
county. 


Montana’s Occupant Protection Programs 


The Montana Department of Transportation uses a number of programs aimed at improving seat restraint 
compliance. This analysis focused on four of its largest efforts: 1) OPI’s driver education program, 2) the 
STEP program, 3) Buckle Up Montana (BUMT) coalitions, and 4) media campaigns. In order to evaluate 
each program’s efficacy, the research team needed to develop a quantitative description of program 
activity. For example, a quantitative metric describing driver education might be school-district-level 
driver education completion rates. For STEP, a quantitative measure could be the number of additional 
hours of enforcement provided. The research team worked with MDT to obtain specific data on each of 
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these programs and to develop program-specific metrics for inclusion in the analysis. Data and metrics for 
each program are described below. 


O NHS/Primary 


County with no 
survey sites 


Figure 3.2. NHTSA seat belt survey sites and average number of responses observed for 2010 to 2012. Point size 
gets larger as the average number of individuals observed at each sampling site increases. 


Driver Education 


In order to examine the impact driver education has on seat belt compliance rates, the research team 
needed a measure of driver education “effort” associated with each NOPUS survey site. After consulting 
OPI representative Fran Penner-Ray, the team compiled school-district-level information on driver 
education completion rates and expenditures using the Montana OPI Growth and Enhancement of 
Montana Students (GEMS) tools (Montana Office of Public Instruction 2010-2012; 
http://gems.opi.mt.gov/ProgramsAndCourses/Pages/TrafficEdSummaryReport.aspx). The GEMS reports 
provided two possible measures of driver education effort: driver education program completion rates and 


average driver education cost per student. 


To assign each site appropriate driver education metrics, the research team overlaid NOPUS seat belt 
survey sites onto a U.S. Census Bureau 2010 map of high school district boundaries (GIS shapefiles 
located here: http://www.landsat.com/montana-free-gis-data.html). The team used the overlaid maps to 
match NOPUS survey sites to local school districts. The GEMS reports are year-specific, so both NOPUS 
surveys collected at a particular site in a particular year were assigned the same value of driver education 
effort. For example, NOPUS site I-01 is located in the Butte High School District, so the driver education 
completion rate and average cost per student associated with both NOPUS survey events at site I-01 in 
2010 come from the GEMS report for the Butte High School District from 2010. Driver education effort 
was treated as missing for sites that fell outside of high school district boundaries. 


After discussions with OPI representative Fran Penner-Ray and MDT, the research team eliminated 
average cost per student as a possible metric of driver education effort, since costs were highly polarized: 
they were fairly low in years when no crashes or other extensive vehicle costs were incurred, and 
substantially higher in years that required vehicle repairs. Since higher driver education expenditures are 
largely driven by equipment issues, as opposed to increased classroom or vehicle time with an instructor, 
it was decided that this metric did not directly reflect driver education effort at the school district level. 


19 


Assessing Montana’s Occupant Protection Programs Quantitative Assessment 


Therefore, driver education completion rates for the local district in the year of sampling were used as the 
sole measure of driver education effort. 


Selective Traffic Enforcement Programs (STEP) 


MDT awards STEP funds to agencies throughout the state. These funds are intended to enhance local 
traffic enforcement programs, and are targeted at high-risk places and times such as college football 
games, summer fairs and festivals, holiday travel times, etc. In Montana, STEP funds are allocated at 
three jurisdictional levels operating at separate geographic scales. Jurisdictions include municipalities, 
counties, and Department of Justice Montana Highway Patrol (DOJ MHP) districts—eight districts, each 
covering multiple counties. Pam Buckman of MDT provided the research team with data describing all 
proposed STEP events that received funding in 2010 (Buckman, 2013). According to MDT, events 
receiving STEP efforts are relatively stable from year to year. For example, if a STEP effort was allocated 
to a particular event in 2010, it was almost certainly allocated to the same event again in 2011 and 2012. 


The research team’s first task was to generate a single quantitative metric describing STEP effort at each 
site. To do this, STEP allocations were aggregated across all three spatial scales at which funds were 
granted. U.S. Census data on municipality boundaries and populations (U.S. Census Bureau 2010 
Geospatial data for urban areas: TIGER product), and county boundaries and populations (U.S. Census 
Bureau 2010 Geospatial data for counties: TIGER product) were used to determine each NOPUS survey 
site’s combination of jurisdictions (e.g., the municipality, county, and DOJ MHP jurisdiction into which it 
fell). For each jurisdiction granted STEP funds, the research team calculated 1) hours of STEP effort, 2) 
number of STEP events, and 3) number of STEP sites within the jurisdiction. This structure does not 
account for the population density or area of a jurisdiction. The research team examined measures that 
accounted for population density and jurisdiction area, but found no relationship between those measures 
and seat belt use rates. 


After determining the municipality, county, and DOJ MHP district in which each NOPUS sampling fell, 
the municipality level STEP effort, county-level STEP effort, and DOJ MHP-level STEP effort at each 
site were added together to obtain aggregated STEP metrics (e.g., hours, number of sites, and number of 
STEP events) for each NOPUS site. Sites that did not fall within a particular municipality were assigned a 
value of 0 for municipality-level STEP effort. 


STEP hours, sites, and number events were highly correlated with one another (see Appendix 3). 
Combinations of predictors that are highly correlated cannot be estimated independently due to an issue 
known as colinearity. Briefly, colinearity is a statistical problem that occurs when estimation techniques 
cannot discriminate between the effect of one predictor and the effect of another similar program 
affecting the same sites. For instance, in this case STEP hours tend to be high for NOPUS sites that are 
also proximal to high numbers of STEP sites and events. Statistical models cannot determine whether 
changes in NOPUS compliance rates associated with increased STEP activity are specifically attributable 
to more STEP hours or more STEP sites, since STEP hours tend to be high for the same sampling events 
that also have high numbers of nearby STEP sites and many STEP events. This is not particularly 
problematic, since the overarching objective of this analysis is to measure STEP effort, and all three 
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metrics (hours, number of sites, and number of events) reflect effort. The research team chose to use 
STEP hours aggregated across all jurisdictional levels as the primary STEP metric in this analysis. 


Buckle Up Montana Coalitions 


In Montana, Buckle Up coalitions typically operate at the county (or multi-county) level. Coalitions do 
not overlap with one another, and not all counties are included in a coalition. Coalition roles include 
written communications and letters to editors of local newspapers, coordination of events and booths 
intended to educate the public about seat belt usage, and synergistic efforts with other programs operating 
in their vicinities. Membership in BUMT coalitions is temporally stable (which is to say, a regions 
remains in the same coalition for multiple years), however new coalitions continue to be formed. This 
evaluation incorporated BUMT coalition activity in two ways. In the first round of assessment, the 
research team compared seat belt compliance rates from NOPUS sampling events located in regions with 
active BUMT coalitions to compliance rates from NOPUS sampling events in counties with no active 
coalition present. Different coalitions were compared to one another in a second round of follow-up 
analyses. 


Media 


Montana Department of Transportation provided reports on major media campaigns coordinated through 
its communications consultants. These reports included information on media markets for each media 
classifications (Television, Radio, Cable, Online, Billboards, Out of home) included in the campaigns, as 
well as the number of paid and earned spots the campaign provided for each media classification. Since 
the media reports provided information on the markets included, the research team was able to spatially 
map each piece of the campaign to a particular region of Montana that it was likely to impact. In this 
assessment, media campaign allocations targeted at a particular municipality were assumed to impact only 
those NOPUS sampling sites within the targeted municipality. 


Media reports provided several potential metrics of media effort, including the number of paid and bonus 
spots procured during each campaign, the total amount paid, etc. As with STEP, the different media 
measures were highly correlated with one another: NOPUS sampling sites in regions with high TV effort 
also typically had high cable, radio, and online effort as well, making it impossible to differentiate 
between impacts of each media platform. To circumvent this issue, the research team used the sum of all 
paid and earned media units as the site-specific measure of overall media effort. 


Incorporation of Potential Confounders 


Seat belt compliance rates are driven by many factors outside MDT’s control. It is a generally accepted 
best practice to incorporate all sources of variation likely to contribute to observed outcomes (as opposed 
to only incorporating those factors that are of explicit interest in the scenario at hand) in a statistical 
analysis. In this analysis, the team included both factors describing the various occupant protection 
programs themselves, and factors like population density, urban/rural status of a given site, and road type 
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that have been historically linked to seat belt compliance rates. If these factors were excluded from the 
analysis, they could mask program effects. For example, assume that higher socioeconomics increases 
seat belt use. If the analysis failed to account for socioeconomic status, the effect of socioeconomic status 
would be attributed to other factors that had similar patterns (for example, urban/rural status; Figure 3.3), 
even though socioeconomic status was the factor actually determining seat belt compliance rates. To 
avoid this kind of misallocation of program effects, the team included several factors historically related 
to seat belt compliance in the analysis. 


Site-level demographic data were obtained using 2010 U.S. Census information whenever possible. 
Population density and local socioeconomic structures have both been shown to be predictive of seat 
restraint usage in past studies (Morgan, 1967; Robertson et al., 1972; CDC, 1986; Hansell & Mechanic, 
1990; Piani & Schoenborn, 1993; Nelson et al., 1998; Schoenborn, 1988; Colgan et al., 2004), and these 
along with weather are thought to be associated with differential patterns of traffic crash severity. 


The models presented here are adjusted for these factors outside MDT’s control. By including these 
factors, the team could draw conclusions about the role different occupant protection programs play, after 
accounting for the presence of other drivers like population density or socioeconomic status. 


2862 63964 125065 


Figure 3.3. Montana counties shaded by population size (on left) and population density (on right) as reported in 
the 2010 U.S. Census. Density is in people per square mile. 


A table of specific factors used this analysis, along with corresponding metadata describing the sources of 
all information included in the model, is available in Appendix 1. All response and factor datasets were 
compiled by constructing a geographic information system (GIS) using the maptools (Bivand & Lewin- 
Koh, 2013), and sp (Pebesma & Bivand, 2005; Bivand et al., 2013) packages in the statistical computing 
environment R (R Core Team, 2013). The research team then “drilled down” through the GIS at each 
NOPUS sampling site and extracted site-specific values associated with each covariate layer in the GIS. 
The resulting dataset of NOPUS sampling events and corresponding covariate values formed the basis for 
the statistical assessment described below. The compiled dataset used to conduct all statistical analyses 
were provided to MDT upon submission of this report. 
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Data Analysis 


To evaluate occupant protection program efficacy, the research team compared seat belt compliance, as 
measured by NOPUS, at sites where the program was implemented (or was implemented more 
extensively) to compliance in sites where the program was absent (or implemented less extensively). 
Because seat belt survey data at each site were collected as a set of counts (individuals observed, and of 
those individuals observed, the proportion wearing seat belts), logistic regression techniques (e.g., 
Agresti, 2012; Gelman & Hill, 2007) were used. The NOPUS data have a distinct spatiotemporal 
structure: sampling events occur at the same sites in multiple years, and multiple times per year, and these 
sites are nested in municipalities, high school districts, counties, and Department of Justice Montana 
Highway Patrol regions. It is important to account for this structure when assessing program impacts 
because the structure can impose unintentional correlations between observed seat belt compliance rates 
in different NOPUS samples. Failing to account for the data collection structure is the same as assuming 
that neighboring sampling sites are driven by independent processes. This would be problematic, for 
example, in a situation where two NOPUS sampling sites that are close together have similar seat belt 
compliance rates, but this similarity is due to some unmeasured factor not included in the model. In this 
analysis, the research team accounted for the hierarchical structure of the seat belt survey data (i.e., 
sampling events occur at sites that are located in specific counties and years) by adjusting for correlations 
between sampling events at the same site with a hierarchical mixed effects model (e.g., Gelman & Hill, 
2007). 


Mixed-effects models with the structure used here involve two different kinds of covariates, referred to as 
fixed and random effects (see Appendix 4: Terminology). Factors that are specifically measured and take 
on a particular value for each site (e.g., the presence or absence of a BUMT coalition, the county-level 
population density for a particular site, etc.) are called “fixed” effects. Relationships between fixed effects 
and seat belt compliance rate are directly estimated in the model, and the statistical significances of those 
relationships are quantified explicitly. Random effects, on the other hand, account for spatially structured 
variation not associated with a particular factor included in the model (i.e., some unidentified and 
unmeasured attribute that makes compliance at one site higher than at another). Statisticians often refer to 
random effects as “soaking up” variation that could not be accounted for otherwise, and that might cloud 
our ability to detect important fixed effects. 


To determine an appropriate set of fixed and random effects, the team fit a series of different models, and 
then performed model selection to identify the effect structure that was best supported by the occupant 
protection dataset. Factors were omitted only if there was no detectable relationship between the factor 
and seat belt compliance rates. Statistical best practices suggest that random-effects structures should be 
identified using a model containing all fixed effects. Fixed effect model selection should occur only after 
the random-effects structure is determined. Therefore, the research team first used likelihood ratio tests to 
identify the support associated with various random-effects structures (e.g., whether a random effect for 
county was sufficient or a site-level random effect was more appropriate; see Appendix 2 for further 
mathematical details) on a model that contained all fixed effects (referred to as the “saturated” model). 
All quantitative factors were standardized (observed value - median) / (standard deviation) prior to model 
fitting to allow for comparison of all factors on a common scale. Model fits were generated using the 
Ime4 (Bates et al., 2013) library in R. Let y;;, be the seatbelt status of the k*" individual observed at the 
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j‘” site during the i*” year. Let X; j be a matrix containing all covariate values associated with the j th site 
during the i‘ year. Let B be a vector of regression coefficients linking the covariate values to the 
response. Let Z; and Z;; be matrices of year- and site-year-specific random effects, respectively. In 
general, the fitted models were of the following form: 


P(vije = 1) = logit~*(XijB + 2;b; + Zijbij); 


exp(XijB+Zjbit Zijbij) 
exp(XjjB+Zjbj+ Zijbij)+V 


P(yijx = 1) = 


where i indexes counties (i = 1, ..., 56) and j indexes over sites (j = 1, ..., 120). Models were fit using the 
total seat belt compliance rates (as opposed to separate models for drivers and passengers), since only 
total values were available in 2011. The general data analysis procedure was as follows: 


Model Preparation 


1. Compare covariates associated with complete cases vs. incomplete cases. The research team 
checked for systematic differences between seat restraint survey samples that were excluded from 
the analysis due to missing information associated with one or several covariates included in the 
model. Any systematic structure to the missing cases was documented to identify sources of bias 
in the model outputs. 


2. Assess colinearity among covariate blocks. To assess colinearity in the MDT occupant protection 
data, the team examined correlations within separate blocks of covariates, with each block 
representing a different MDT occupant protection program. When multiple metrics describing the 
same program were highly correlated with one another, only a single measure was included in the 
assessment models. 


3. Identify the appropriate random-effects structure. The team fit models with all fixed effects and 
different random-effects structures (site, county, observer), including a nested structure where 
sites were nested in counties. Restricted error maximum likelihood (REML) methods were used 
to identify an appropriate random effects structure. 


Model Fitting 


1. Fit models with all non-MDT fixed effects. Models with various fixed effect combinations were 
fit using the random-effects structure identified in (3), but with only those fixed effects that had 
not been manipulated by MDT (i.e., only fixed effects not related to occupant protection 


programs). Fits were obtained through standard maximum likelihood methods, and model 
selection was conducted using likelihood ratio tests to compare models with nested fixed-effects, 
and AIC-based methods for models with non-nested fixed effects. 


2. Examine the role of specific occupant protection programs individually. The team examined how 
residuals from the model using the random-effects structure identified in Model Preparation, Step 
3, and the non-MDT fixed-effects structure identified in Model Fitting, Step 1 related to activity 
of various MDT programs. The researchers graphed all relationships, and then used standard 
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maximum likelihood estimate techniques for model fitting. Model selection was conducted using 
likelihood ratio tests in settings where models were nested, and AIC-based methods in cases 
where they were not. This step illustrated relationships between each MDT program and seat 
restraint use rates, but did not account for the presence of other MDT programs. 

3. Fit a saturated model with all MDT and non-MDT terms. The final model included all MDT 
program factors and all non-MDT factors. This model estimated each program’s impact on seat 
restraint use after accounting for the influence other MDT programs might have at each site. 


4. Expand to special cases. Several relevant subsets of data that were excluded from initial fitting 
were examined to determine the extent to which adding those subsets influences model 
coefficient estimates. Foremost among these was the inclusion of the three seat belt survey sites 
located on tribal lands. Different structures of several MDT-related predictor variables were also 
considered at this stage (e.g., whether BUMT-coalition-specific predictors were necessary). 
Conduct follow-up assessments contrasting occupant protection program implementation in 
counties with particularly good vs. particularly bad outcomes. 


Model Preparation Results 


Comparison of complete and incomplete cases 


The final covariate matrix used for the analysis consisted only of sites with no missing values for any 
MDT predictor. Of the 720 seat belt survey samples available, 110 had missing values for one or more 
predictors included in the model, 
and were therefore eliminated. The 
consultants checked for systematic 
differences between sites with 
missing values and sites that were 
included in the analysis to identify 
potential sources of bias. Removal 
of sites with missing data resulted 
in complete elimination of the 
following counties from the 
analysis: Broadwater, Granite, 
po Missing program information Phillips, Pondera, Powell, 

Roosevelt, and Valley (see Figure 
3.4). These counties typically had 
smaller populations than those 
included in the analysis (median 
Figure 3.4. County-level inclusion status. county population is 


iz Full (non-missing) program information 


approximately 5,000 for removed 
counties vs. 30,000 for included counties) and represent less than 5 percent of Montana’s population. 
Removed sites were also spatially structured, representing the northeastern portion of the state as well as 
several western counties. Eliminated cities included Townsend, Amsterdam, Malta, Conrad, Wolf Point 
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and Glasgow. Missing OPI driver education completion rates were the most common reason that a 
sampling event was eliminated from the analysis. The municipalities of Glasgow and Amsterdam were 
also missing STEP information, and were excluded from the analysis. 


Investigation of colinearity 


The team examined colinearity between different metrics for media activity, and different STEP metrics. 
In both cases, all measures were strongly correlated with one another. Therefore, only a single media 
metric and a single STEP activity metric were included in the analysis (see Appendix 3). The following 
fixed effects were included in the model suite: stratum, month and year of sampling, mean temperature on 
day of sampling, aggregate precipitation on the day of sampling, sampling day, local county population 
density based on county-level population size in the 2010 census, median county income, an aggregate 
measure of local STEP hours invested per person per square mile locally, the completion rate for local 
OPI driver education programs, a categorical variable for the presence of a local Buckle Up Montana 
coalition, and a sum of local media investments. The Montana Department of Transportation was 
particularly interested in the effect of overlaying multiple programs in a spatiotemporal domain, so the 
research team also included second-order interaction terms linking all predictors related to occupant 
protection programs, which provide insights into whether programs operate synergistically or they do not 
work as well in the presence of other programs. Models that included sampling day failed to converge, so 
it was eliminated at the outset of the modeling procedure. 


Identification of appropriate random-effects structure 


The research team tested the importance of including both site-level and county-level random effects 
using a likelihood ratio test (which was likely to yield conservative results in this case; see Appendix 2). 
The likelihood ratio test indicated that site variation was non-negligible (= 554.8 on 1 degree of freedom; 
p-value < 0.0001), thus random effects for both site and county were included in the model. Then, a non- 
MDT fixed-effects model using county-level random effects was fit to initiate investigation of the 
model’s fixed-effects structure, (see Table 3.1 for the fixed-effects estimates from the non-MDT model). 


Identification of appropriate fixed-effects structure 


The urban/rural indicator and all interactions involving STEP or driver education did not improve model 
fit, so these factors were eliminated from the model. The final model retained all main effects, both from 
MDT programs and non-MDT drivers, as well as a quadratic effect on media. Inclusion of MDT program 
predictors significantly improved model fit (see Table 3.2), suggesting that MDT program activity is 
associated with seat belt use rates in Montana. While all programs were related with increased compliance 
when examined without accounting for the presence of other programs (Figure 3.6), some of these effects 
were no longer detectable when all programs were examined in a single model (Figure 3.7). This 
underscores a weakness in the dataset: often, multiple programs impact the same set of NOPUS survey 
sites, making it impossible to identify the impact individual programs have on seat belt use. The specific 
estimates of each program’s impact are shown in Figure 3.6 and 3.7, and statistically described in Table 
3.3. 
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Model Interpretation 


Logistic regression models like the ones used here relate changes in the odds that a person uses his or her 
seat belt (in this case, based on local NOPUS seat belt use surveys) with changes in particular predictor 
variables, or “factors”. The logistic regression model estimates an exponentiated coefficient that describes 
the effect each predictor included in the model has on the likelihood that an individual wears his or her 
seat belt. This exponentiated coefficient describes the multiplicative change in the odds that an individual 
uses his or her seat belt when the predictor is present, relative to a baseline scenario where the predictor is 
absent. In Figures 3.5 and 3.7, the multiplicative change in odds is written as “equally likely”, “.25x as 
likely”, “4x as likely”, etc. For effects overlapping the “equally likely” vertical line, the model detected 
no difference between likelihood of seat belt use for people who were exposed to a particular program 
(e.g., who received the “treatment”) and people in the baseline group. In the “.25x as likely” and “4x as 
likely” cases, % the likelihood of wearing their seatbelts, or were four times as likely as the baseline 
group to wear their seatbelts, respectively. 


In this analysis, the baseline model is based on data from NOPUS surveys conducted in 2010 at sampling 
sites located on city roads. Note in Table 3.2 that the estimated effect of Interstate is to the right of the 
“equally likely” line. This means that Interstate travelers were significantly more likely to wear their 
seatbelts than drivers on city roads. To determine the specific odds of compliance at an Interstate site, 
take the baseline odds, and multiply them by the exponentiated coefficient associated with Interstate. In 
the Interstate situation, this means that the odds of seat belt use at Interstate sampling sites are greater- 
than-one time the odds of compliance at sampling sites located on city roads. The odds of compliance at 
sampling events with predictors that have exponentiated coefficients less than one are lower than the odds 
of compliance at the baseline site. 


All continuous predictor variables included in this analysis (temperature, precipitation, population 
density, median population income) were standardized. Standardization is a method of rescaling each 
continuous variable so that its effect on seat belt use is estimated relative to its overall variation. This 
makes the estimated effect of population density (which varies massively across the study area) on seat 
belt use comparable to the estimated effect of total precipitation on the day of sampling (which takes on 
values over a much narrower range). The exponentiated coefficients associated with the continuous 
predictor variables shown in Table 3.2 reflect the change in odds of seat belt use associated with 
increasing that particular covariate from its median value to approximately its 85" percentile. For 
precipitation, this estimate reflects the change in odds associated with increasing precipitation from a 
trace amount to one-tenth of an inch; the exponentiated coefficient for population density is equal to the 
change in odds associated with increasing the population from the median population density (15.5 
persons per square mile) to the 85" percentile of observed population densities, which is 36 persons per 
square mile. 


Examination of non-MDT fixed effects. 


The models showed substantial differences in seat belt compliance rates between the four NHTSA seat 
belt survey strata. Compliance at City sites was significantly lower than at Interstate, Primary, and 
Secondary/County sites (see Table 3.2, and Figure 3.5). In general, compliance was slightly (but 
significantly) lower in 2011 and 2012 than in 2010, consistent with MDT’s concerns regarding the 
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potential for declining seat belt use rates in Montana. The model accounted for year-specific differences 
in compliance rates when evaluating all other predictors. Compliance was slightly, but statistically 


Table 3.1. Non-MDT model fixed-effect estimates. P-values indicate the significance of the relationship between 
each fixed effect and seat belt compliance rates, with smaller p-values indicating stronger associations (p-values of 
0.05 are often used as a cut-off for statistically significant effects). Positive estimates are associated with increased 
compliance with increased values of the fixed effect. The significant negative interaction between total media and 
BUMT presence suggests that while both increased media and BUMT presence are associated with increased 
compliance, when both are present, those effects cancel each other out to some extent. All results are relative to a 
baseline model built on April 2010 data from the Butte—Silver Bow BUMT coalition. 


*Reported standard errors are associated with raw coefficients, prior to exponentiation. 


Coefficient Exp(Coefficient) Std. Error* Z-value p-value 
Intercept 

(Intercept) 1.558 0.08595 4.952 <0.001 
Strata 

Interstate 3.314 0.1303 9.195 <0.0001 

NHS Primary 2.960 0.1111 9.762 <0.0001 

Sec County 1.305 0.08740 3.042 0.002352 
Year 

2011 0.8709 0.02086 -6.626 <0.0001 

2012 0.8336 0.02017 -9.025 <0.0001 


County Demographics 
Co. Median Income standardized 1.302 0.07711 3.422 <0.0001 


Co. 2010 Population Density 1.408 0.09211 3.718 <0.0001 


Sampling Event factors 
Precipitation 0.9684 0.007404 -4.343 <0.0001 
Mean Temperature 1.045 0.008890 4.963 <0.0001 


significantly, lower on days of precipitation than on dry days, and lower on hot days than cooler ones. 
Compliance was significantly higher in June than in April/May, although this effect is likely driven in 
part by the timing of MDT spring media campaigns. The model also showed that county population 
density and median income were both significantly associated with higher seat belt compliance rates. 


Table 3.2. Likelihood ratio test results comparing Non-MDT model to model with MDT predictors. 


Model Parameter change from Test statistics (degrees P-value AIC 
full model of freedom) 

Non-MDT - -- - 4734.1 

MDT 5 17.25 (5) 0.004 4726.9 
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Interstate 

NHS/Primary 
Secondary/County 
Precipitation 

2011 

2012 

Mean Temperature 
County Population Density 
County Median Income 


4 2 1 2 4 


Multiplicative change in odds of seat belt use 


Figure 3.5. Model coefficients associated with all factors included in the non-MDT model. The plot shows the 
multiplicative change in odds of seat belt usage when moving from the baseline group to a modified group that 
includes each predictor. Factors with horizontal lines overlapping one have no significant impact on occupant 
protection; those in excess of one are associated with significant increases in seat restraint usage, and those less 
than one are associated with diminished seat restraint usage. The baseline model describes seat belt compliance in 
2010 on City roads, at the median temperature, observed temperature and precipitation levels, in a county of the 
median population size and income in April, with no MDT programs present. Thin horizontal lines reflect 95 
percent confidence intervals around each effect’s odds ratio; thick horizontal lines are 50 percent confidence 
intervals. 


Individual examination of MDT's occupant protection programs 


This section describes relationships between each MDT program and seat belt use after accounting for the 
non-MDT factors described previously. Relationships described in this section do not account for 
multiple programs simultaneously influencing the same site. Those results are presented in the next 
section. 


There was a significant increase in seat belt compliance associated with increasing STEP activity (upper 
left panel, Figure 3.6) when STEP was examined in the absence of other programs (see Recommendation 
1). This pattern was less pronounced at sites in rural counties than at sites in urban ones (upper right 
panel, Figure 3.6), likely reflecting differences in the total number of drivers that passed STEP patrols. 


Sites near active Buckle Up coalitions had significantly higher compliance rates than did sites 
away from Buckle Up coalitions (lower right panel, Figure 3.6). 


All media metrics were highly correlated with one another, making a comparison of the associations of 
particular media with seat belt compliance problematic (see Appendix 3, and discussion of colinearity in 
the Data Preparation section above). To circumvent this, the research team used an aggregated media 
measure that reflected all media employed. Analysis of this aggregate metric suggested that additional 
media expenditures were significantly associated with increased compliance, up until about $12,000 of 
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investment (lower left panel, Figure 3.6). At that point, media impacts plateaued. This analysis is limited 


by the geographic extent of the media campaigns, however, and merits further investigation (see 
Recommendation 2). 
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Figure 3.6. Relationships between residuals from the Non-MDT model and individual MDT programs. Residuals 
capture variability not accounted for in the model. Relationships between residuals and MDT program activity 
suggest that MDT programs are associated with changes in seat belt compliance rates. Program activity is 
quantified following the description in the Data Preparation section. This visualization reflects the same 
relationships detailed in Table 3.4, but in the absence of random effects. Statistical significance from the model 
shown in Table 3.4 is consistent with the direction and strength of these relationships, however the model formally 


incorporates correlations due to replicate sampling at the same sites and counties, which cannot be accounted for 
in this figure. 
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Interstate 

NHS/Primary 
Secondary/County 

County Population Density 
County Median Income 
Post Media Campaign 

2011 

2012 

Mean Temperature 
Precipitation 

Media $ with Buckle Up MT 
Media $ no Buckle Up MT 
Buckle Up MT Coalition Present 
Aggregate STEP Hours 
Driver's Ed Completion Rate 


.25x as likely Half as likely Equally likely Twice aslikely 4x as likely 


Odds of seat belt use if covariate is increased one unit, relative to baseline group 


Figure 3.7. Model coefficients associated with all factors included in a model containing both MDT and non-MDT 
drivers of seat restraint usage. The plot shows the change in odds of seat belt usage when moving from the 
baseline group to a modified group that includes each predictor. Factors with horizontal lines overlapping the 
vertical “equally likely” line no detectable impact on occupant protection; those to the right of the “equally likely” 
line are associated with significant increases in seat restraint usage, and those to the left of “equally likely” are 
associated with diminished seat restraint usage. The baseline model to which each effect is compared describes 
seat belt compliance in 2010 on City roads, at the median temperature, observed temperature and precipitation 
levels, in a county of the median population size and income in April, with no MDT program activities. Thin 
horizontal lines are 95% confidence intervals for each effect; thick horizontal lines are 50% confidence intervals. 
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Table 3.3. Coefficient estimates from full model that includes both non-MDT and MDT-program-related predictors 
of seat belt compliance. Significant (i.e., p-value <= 0.05) positive (i.e., coefficient value > 0) relationships indicate 
programs associated with significant increases in seat belt compliance rates. 


Coefficient Exp(Coefficient) Std. Error* Z-value p-value 
Intercept 

(Intercept) 1.19 0.14 1.29 0.20 
Strata 

Interstate 3.73 0.13 10.19 <0.001 

NHS Primary 3.10 0.11 10.27 <0.001 

Sec County 1.40 0.09 3.78 <0.001 
Year 

2011 0.892 0.03 -4,.15 <0.001 

2012 0.835 0.02 -4,51 <0.001 
County 


Demographics 
Co. Median Income 
standardized 1.22 0.07 2.96 <0.001 
Co. 2010 Population Density 1.31 0.08 3.34 <0.001 


Sampling Event 


factors 
Mean Temperature 1.05 0.01 5.12 <0.001 
Precipitation 0.969 0.01 -4.15 <0.001 
MDT Programs 
OPI Completion Rate 0.942 0.05 -1.12 0.26 
Total Media Cost 0.999 0.00 -2.21 0.03 
Total Media Cost Squared 1.00 5.77 x e-10 2.09 0.04 
Summed STEP Hours 1.00 0.0002 3.16 0.002 
BUMT Present 1.15 1.46 0.93 0.35 


MDT Interactions 


TotMediaCost : BUMT Present 2.53 0.0153 -4.8707 <0.0001 


*Reported standard errors are associated with raw coefficients, prior to exponentiation. 


Simultaneous examination of MDT’s programs 
Driver Education 


Driver education completion rate had no detectable relationship with seat restraint usage rates (Figure 3.7; 
Table 3.3). This could be because driver education was widely available, and seat restraint usage was only 
one of many aspects emphasized in the current driver education curriculum. The Montana Department of 
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Transportation recently collaborated with OPI to update its traffic education curriculum to include a 
special segment that focuses on the importance of seat belt use by teens. This update may impact the 
influence of driver education on occupant protection in the future. 


STEP 


Although STEP was associated with increased seat belt use when examined on its own (Figure 3.6A, 
3.6B), that effect was no longer detectable in a model that accounted for all programs (Figure 3.7; Table 
3.3). While this provides some weak evidence that STEP may be effective at increasing seat belt use, that 
effect cannot be clarified with the existing data for two reasons. First, STEP efforts often overlap with 
other program activity, and second, STEP efforts intentionally target events where individuals may be 
more likely to engage in risky behaviors. As a consequence, even if STEP efforts do improve seat belt use 
rates, that effect could be masked because STEP operates at events where seat belt use may well be lower 
than average. STEP may in fact improve compliance by raising very low seat belt use up to a moderate 
level, but that impact cannot be determined from these data. As a consequence, the team recommends 
follow-up investigations aimed at isolating the impacts of STEP (Recommendation 1, page 38). 


Buckle Up Montana Coalitions 


Buckle Up Montana coalitions were generally associated with increased compliance (Figure 3.6D; Figure 
3.7; Table 3.3). However, coalitions differed in the magnitude of this effect (see Follow-up Analysis). 
Furthermore, the impact of BUMT was diminished in cases with higher media intensity, suggesting that 
these two programs may be redundant with one another. 


Expansion to special cases 


A follow-up analysis was used to allow for comparison between different BUMT coalitions. This analysis 
found strong evidence that different BUMT coalitions are associated with different seat belt compliance 
rates (even after accounting for county-level differences in population size and median income, road types 
and weather conditions sampled, media, and local STEP activity; see Figure 3.8). The Ravalli County, 
Lake County, and Greater Gallatin coalitions were associated with particularly high seat belt use rates, 
after accounting for all other factors. Other coalitions had similar associations with observed seat belt use 
rates. 
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Dawson County 
Flathead County | 
Greater Galtin County 
Hill County | 
Lake County | 
Missoula County | _ 
PT & T County | 
Ravalli County | 


Tri County 
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Multiplicative shift in odds of seat belt usage, relative to Butte-Silver Bow Coalition Sites 


Figure 3.8. Unique fixed effect associated with each Buckle Up Montana coalition. All effects are estimated relative 
to seat belt compliance rates at NOPUS sites located in the Butte-Silver Bow County coalition. The plot shows the 
multiplicative change in odds of seat belt usage when moving from a Butte-Silver Bow site to a site located in the 
labeled coalition’s jurisdiction. Factors with horizontal lines overlapping one are not significantly different from 
Butte-Silver Bow; intervals in excess of one are associated with significant increases in seat restraint usage in the 
labeled coalition (for example, Ravalli County coalition sites have significantly higher compliance than Butte-Silver 
Bow sites). All comparisons are made using a model that has adjusted for county population density, median 
income, precipitation and temperature, as well as other MDT occupant protection program effects. 


A second follow-up analysis was used to examine how robust the models were to the exclusion of the 
driver-education-related predictors. Missing driver education values accounted for 98 of the 110 
incomplete data points (see Data Preparation section above). In this follow-up analysis, the research team 
fit a model identical to that presented in Table 3.3 but without driver education completion rates. 
Program-specific effect estimates from this new model were compared to estimates from a model that 
included only the complete cases. If program effects for incomplete cases are similar to effects for 
complete cases, then the effect estimates from these two models should be similar. The analysis suggested 
that other programs performed similarly at sites missing driver education data and at sites where driver 
education was available. 


Discussion 


Utility of multiple programs 


Interaction terms were used to test whether the simultaneous effect of two programs differed from the 
sum of the effects of each program individually. The results of this analysis support a significant negative 
interaction between BUMT presence/absence and total local media expenditures. This suggests that the 
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simultaneous effect of these two programs in a given county is less than the sum of their individual 
effects. The absence of significant interaction terms between other programs suggests that all other 
program combinations generally have additive effects. This means that the impact that one program has 
on seat belt compliance rates does not depend on whether other programs are present. Therefore, using 
multiple programs in the same region should increase local seat belt compliance; when new programs are 
added to a region, their positive impact is added to the effect of other programs already operating in that 
vicinity. As a consequence, this model suggests that using multiple programs in a particular vicinity is 
generally beneficial (see Recommendation 1, page 38). Media campaigns and BUMT coalition presence 
were the only exception to that rule detected in this assessment. The relative impact of Buckle Up 
Montana coalition presence declined in the presence of additional media activity. Having two programs 
present is still superior to only one program, but the impacts of both programs together are slightly less 
than the sum of the two programs’ effects when operating alone. 


Low compliance groups and regions 


Compliance was significantly lower in cities and secondary/county roads than on Interstates or 
NHS/primary highways. Lower compliance rates were also associated with low county population sizes 
(increased compliance with increased population size), and lower median county income levels. 


Analysis caveats 


While the research team invested a great deal of effort in ensuring that this assessment leverages the best 
available data, data-based limitations still remain. The spatial scales represented in the models used for 
this assessment assume strong internal mixing within counties, and also assume that county-level resource 
allocations impact all parts of a county equally, whereas in reality, they are likely to scale with local 
population densities. Furthermore, individual choice about seat restraint usage is driven by numerous 
factors, only a few of which were available for incorporation into this analysis. These results rest on the 
assumption that the majority of seat restraint choice determinants are included among the model 
covariates (and this set of factors was informed by existing research on seat restraint choice), however it 
is entirely possible that some factors that contribute to seat belt usage are not accounted for here. 
Additionally, response data are sometimes spatiotemporally separated from program-specific data, which 
limits this model’s power to detect true effects of MDT occupant protection programs. While the team is 
confident that effects detected in this assessment are real, the failure to detect an effect of a given program 
could stem from limited statistical power, and should therefore be viewed as grounds for further 
investigation. To improve model strength and more precisely identify program impacts on seat belt use, 
MDT may want to consider collecting additional data that isolate the effects of specific programs (see 
Recommendation 3, page 38). 


Association does not imply causation 


Even the strongest associations uncovered in this analysis do not imply causal relationships. Specifically, 
observing that NOPUS samples with high compliance occur in the presence of a particular predictor does 
not mean that the factor of interest caused the observed higher compliance. This limitation is universal 
when analyzing observation data (as opposed to data from designed experiments). It is particularly 
important to take this into account when considering differences between the Buckle Up Montana 
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coalitions, and when assessing the inconsistent relationship between STEP and seat belt compliance rates. 
It is likely that other factors are at play in each county, and that these factors change the starting point 
from which each coalition operates. Similarly, as noted in the Results section, the apparent lack of an 
effect for STEP in the model that contained all programs (e.g., Figure 3.7; Table 3.3) may reflect that 
STEP efforts are allocated to events where seat belt use might otherwise be very low. While this analysis 
provides some guidance about associations between program activity and seat belt use, causal linkages 
could only be established through designed experiments, where some sites were randomly chosen to 
receive programs, while others were not (see Recommendation 3, page 38). 


Conclusions 


MDT programs contribute to a small increase in seat belt use rates, but their 
impacts are small relative to other factors 


The drivers with the largest effect of seat belt use rates in Montana are road type, population density, and 
income (Figure 3.7). Since none of these factors are under MDT’s control, it would be easy to overlook 
the role that MDT’s programs play. However, the effects associated with several MDT programs were 
significant, albeit relatively small. Buckle Up Montana coalition presence and media presence were both 
associated with increased seat restraint usage (though this increase was diminished when both programs 
impacted the same site). The slightly negative, but statistically insignificant relationship between driver 
education completion and seat belt use is likely to change as Montana’s OPI reworks its driver education 
curriculum. 


STEP may be highly effective, but its impacts are masked by the presence of other 
programs 


STEP was the most effective of the programs examined here when analyzed in the absence of other MDT 
programs (see upper left and right panels of Figure 3.6), but that impact eroded when other programs 
were included in the model (see Figure 3.7 STEP). This is likely a feature of the data used here, and may 
not reflect a true absence of a STEP effect. Among the programs examined here, STEP would benefit 
most from additional data collection (see Recommendation 3, page 38). STEP activity had similar impacts 
on seat belt use regardless of the population density or spatial jurisdiction in which they were allocated. 
This means that additional STEP hours allocated to a very rural county had similar impacts on seat belt 
use as STEP hours applied in an urban setting. In other words, STEP effort in urban and rural 
jurisdictions must access roughly the same number of vehicle passengers. Therefore, rural jurisdictions 
must be allocating STEP very effectively toward target periods of intense vehicle travel. 


Media investment has initial benefits, but benefits decline when expenditures 
exceed $12,000 


A significant quadratic effect characterized the relationship between additional media investment and seat 
belt usage when media was examined alone (lower left panel of Figure 3.6). This suggests that although 
initial media investments result in substantial gains in seat belt use, after a certain point, additional dollars 
no longer elicit the same benefit. 
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BUMT is highly effective, but BUMT and media are most beneficial when they 
operate separately 


BUMT had the strongest effect of any MDT program analyzed here, when all programs were considered 
in the same model (see Figure 3.7, Buckle Up MT coalition present). The slightly antagonistic 
relationship between media expenditures and BUMT presence suggests that perhaps BUMT coalitions 
and media campaigns overlap in their target audience and target method of increasing compliance (see 
Figure 3.7, Media $ with Buckle Up MT vs. Media $ with no Buckle Up MT). Efficiency might improve 
if the BUMT/media relationships were better coordinated (thus eliminating redundant effort). 
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4. Recommendations and Conclusions 


This project produced direct estimates of program impacts on seat restraint usage in Montana. The 
findings provide an additional line of evidence supporting the general efficacy of MDT's occupant 
protection programs. This project resulted in no implementation plan or timeline; instead, the findings 
presented here are intended to help MDT make informed decisions about program continuation. To that 
end, the research team respectfully puts forth the following recommendations to MDT regarding resource 
allocation to state occupant protection programs. 


Recommendation 1 


Continue existing Selective Traffic Enforcement Program and Buckle Up Montana coalition efforts, since 
these programs were individually associated with increased seat restraint usages in their target vicinities 
(Figure 3.6). The existing data suggest that STEP impact may decline in the presence of other programs 
(Figure 3.7). To clarify STEP’s specific impact, consider running a designed experiment to isolate the 
effect of STEP from other MDT programs. 


Recommendation 2 


In this dataset, media investment was associated with increased seat belt use at proximal NOPUS survey 
sites, but this was only true for sites that did not also have a local BUMT program (Figure 3.7). 
Furthermore, an individual assessment of media effects suggested that while the first $12,000 of media 
investment are very effective at increasing compliance, further investments have diminishing returns 
(Figure 3.6). Both of these trends need further investigation. Media investment was very similar 
throughout the state, which clouded estimates about the per-capita, per-dollar benefit of media campaigns. 
Consider diversifying the size of media investments so that a range of different media investment values 
can be explored. Additionally, it would be beneficial to run an experiment in which media and BUMT 
effects could be separated. This might involve randomly assigning the six largest Montana municipalities 
to three different treatment groups: BUMT-only, media-only, or BUMT + media, applying each treatment 
for one year and measuring the NOPUS seat belt use rates for each municipality, and then re-assigning 
cities to different treatment groups in a second year, measuring NOPUS use rates then. This design would 
allow for a very clear assessment of the separate and combined benefits of BUMT and media campaigns. 


Recommendation 3 


MDT should consider supplementing NHTSA's NOPUS data survey with program-specific data 
collection that better isolates the impacts of particular programs through pre-intervention versus post- 
intervention data collection and analysis. For example, MDT might collect pre- and post- drivers 
education data on compliance in high school parking lots, or collect before and after sampling of STEP 
sites to see if interventions appear to change rates on a fine scale. Also, program impacts can be estimated 
more precisely in the presence of strong baseline data recorded prior to program implementation. 


Recommendation 4 


MDT should consider moving all of its data records to an electronic format, as this will lower costs and 
improve efficiency in future program evaluations. 
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Recommendation 5 


This analysis suggests that program effects are additive, with the possible exception of media and BUMT. 
Therefore, it would likely be advantageous to invest in multiple programs in a county (with the possible 
exception of BUMT and media). 


Recommendation 6 


For a more comprehensive approach beyond the data driven approach described here, it is critical to 
obtain driver’s attitudes and behaviors regarding seat restraint usage. It is recommended that a follow-up 
human factors assessment of Montana’s drivers be conducted to understand the current traffic safety 
culture; with the aim to identify strategies to increasing seat restraint usage across the state. This would 
include recruiting a diverse sample of drivers across the state to collect relevant quantitative and 
qualitative data through user surveys and focus groups. 
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Appendix 1. Model Covariate Sources 


General Covariates 
Covariate Specific Quantity Used (with resolution) 
NOPUS-related Covariates 


Site Categorical variable with levels corresponding to each 
seat belt survey sampling event 


Stratum Categorical variable with four levels: Interstate, NHS 
and Primary, City, Other (Secondary and County) 


Month Categorical variable with two levels: April, June 
Day Categorical variable with seven levels 
Time Quantitative variable ranging from 7:00 AM to 7:00 PM 


MDT Occupant Protection Program Covariates 
Year Categorical variable: 2010, 2011, 2012 


Precip Quantitative variable for amount of precipitation on day 
of sampling in proximity to site 


Temperature Quantitative variable describing average temperature 
on day of sampling in proximity to site 


Geographic Covariates 


County County within which each survey site falls 
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Source Description 


NOPUS 


NOPUS 


NOPUS 


NOPUS 
NOPUS 


NOPUS 


Weather 
Underground site 
data. See full 
description of data 
extraction protocol in 
Data Prep Appendix 


Weather Under- 
ground site data. See 
full description of 
data extraction 
protocol in Data Prep 
Appendix 


U.S. Census Bureau 
2010 TIGER 
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Covariate 
School District 


Urban/Rural 


Tribal Status 


DOJ District 


County population 


County Median 
Household Income 
County Percent without 
Health Insurance 


BUMT coalition indicator 


BUMT coalition 


BUMT coalition annual 
budget 


City STEP Days 


City STEP Hours 
City STEP Sites 


County STEP Hours 


Specific Quantity Used (with resolution) 
School District within which each site fell 


Categorical variable for the1-8 district a site falls in 


Demographic Covariates 


Quantitative variable: Population for a given site's 
county 


Quantitative variable 


Quantitative variable taking on values between 0 and 1 


Occupant Protection Program Covariates 


Indicator variable for whether a BUMT coalition was 
present 


Categorical variable with separate levels for each BUMT 
coalition, and NA if no coalition was present in a site's 
county 


Quantitative variable Reflects either budgeted re- 
sources or expended resources, depending on available 
information 


Quantitative; assumed the same for all years 


Quantitative; assumed the same for all years 
Quantitative; assumed the same for all years 


Quantitative; assumed the same for all years 
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Source Description 


US Census Bureau 
2010 TIGER 


US Census Bureau 
2010 Geospatial data 
MT Department of 
Justice Website 


U.S. Census Bureau 
2010 


American Community 


Survey, 2006-2010 


American Community 


Survey, 2006-2010 


Pam Buckman 


Pam Buckman 


Coalition directors 


Pam Buckman 


Pam Buckman 
Pam Buckman 


Pam Buckman 
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Covariate Specific Quantity Used (with resolution) Source Description 
County STEP Days Quantitative; assumed the same for all years Pam Buckman 
County STEP Sites Quantitative; assumed the same for all years Pam Buckman 
DOJ District STEP Days Quantitative; assumed the same for all years Pam Buckman 
DOS District STEP Sites Quantitative; assumed the same for all years Pam Buckman 


Local Media Impact 


Quantitative variable equal to Students Enrolled / Fran Penner-Ray / 
OPI Drivers Ed Students Complete; OPI data matched to local school GEMS* tool 
Completion Rate district and year for each surveying event 
OPI Average Cost per Quantitative variable; OPI data matched to local school Fran Penner-Ray / 
Student district and year for each surveying event GEMS* tool 


Table A1.1. Covariates considered for inclusion in MDT occupant protection program assessment. *Montana Office 
of Public Instruction, 2010-2012. 


Covariate Source Descriptions 


In order to analyze the efficacy of Montana's various occupant protection programs, we constructed a 
compiled dataset containing seat belt compliance information (“Response Data") collected in conjunction 
with the NOPUS NHTSA seat belt surveillance efforts, along with data on driver education, media 
expenditures throughout the state, etc. Detailed meta-data for each source is included in the following 
sections. 


Response Data 


Data were acquired through MDT from seat-belt surveillance exercises conducted in conjunction with 
NHTSA in April and June of 2010, 2011, and 2012. Response data were available for 120 sites located in 
one of four classes of road (Interstate, primary road, city road, secondary/county road). Responses were 
available for both drivers and passengers in 2010 and 2012, but not in 2011. Number of lanes was also 
recorded. 


Tribal Tract Data 
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Boundary shapefiles for Montana tribal tracts according to the 2010 U.S. Census were extracted by KRM 
on 25 June 2013 from the following site: http://www.landsat.com/montana-free-gis-data.html. We 
followed the link to Tribal Tracts to access the U.S. Census Bureau's 2010 TIGER product for tribal tract 
boundaries. By default, TIGER products are projected following the Global Coordinate System North 
American Datum 1983 (GCS NAD83) system. TIGER metadata are available through the U.S. Census 
Bureau, and are in compliance with both the Census Bureau Geospatial Product Metadata Standard, and 


the Federal Geographic Data Committee Content Standard for Digital Geospatial Meta-data. 


Socioeconomic Data 


We used data from the American Community Survey 2006-2010 on county level median household 
income. Information on median household income was accessed by K. Manlove on 13 June 2013 from the 
site http://ceic.mt.gov/IncomePage.aspx. 


Urban/Rural 


Use data under Montana City Boundaries link on the page 
http://gisportal.msl.mt.gov/geoportal/catalog/main/home.page 


Weather data 
http://www.wunderground.com/history/airport/KL VM/2013/8/23/DailyHistory.html 


Protocol: 
1. Use seat belt calendar.xlsx doc and Seatbelt Count Sites.doc from Rebecca Goodman. 


2. In wunderground.com’s search engine, search location indicated at top of Seatbelt CountSites.doc 
description for each site. If no location is indicated, use last location indicated for previous sites 


3. Scroll down to Weather History and Almanac. 
4. Select the date on which sampling occurred. 
5. Scroll down to the hourly weather table 


6. Extract and average daily temperature and precipitation data that correspond most closely in time to the 
sampling time indicated in Seatbelt Count Sites.doc. We used closest town for which the site specified 
(usually this meant the town last mentioned to find the site for data collection. We inputted the actual 
mean temperature in Fahrenheit from weather underground (wunderground.com) and the amount of 
precipitation in inches to the nearest hundredth. Some sites had precipitation measurements of trace 
amounts; these were coded as 0 for simplicity. Historic weather information could not be extracted for 
smaller locations, so nearby centers were chosen to appropriately represent them (these were chosen by 
smallest distance and knowledge of geography). The list below indicates the urban center used to obtain 
weather data for the listed sites: 


Kalispell: S-05, P-15, S-17, C-01, C-03, S-06, P-04, C-02, P-03, C-04, S-18 
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Drummond: P-10, P-20, P-06, P-09 

Missoula: C-08, I-06, S-09, P-05, C-09, S-03, C-07, C-06, C-05, C-10, I-05, S-02 
Helena: C-26, C-28, S-08, S-19, P-22, P-24,C-36, C-29, C-27, I-02 

Bozeman: S-07, I-07, P-14, C-39, C-43, C-40, C-44, C-42, C-41, S-20 
Livingston: C-45, P-08, C-46, C-47, P-19,1-08 


Billings: P-11, C-50, C-56, C-48, S-01, C-49, C-52, C-51, S-15, C-53, C-58, C-57, C-55, C-54, I-09, S- 
10, C-60, C-61, I-10, I-11 


Butte: C-37, I-01, P-13, C-38, S-14 

Great Falls: C-20, S-04, C-24, C-18, C-22,C-19, I-03, S-16, C-23, P-07, S-11, S-21, C-17, C-25 
Havre: C-15, C-13, C-14, P-21 

Cut Bank: C-16, C-11, S-13, C-12, P-01, I-04 

Miles City: C-64, C-63, C-62 

Glendive: I-12, C-32, C-33, P-18 

Wolf Point: C-31, P-02 

Glasgow: P-12, C-30, C-59, P-23 

Lewistown: P-17, C-34, S-12, C-35, P-16 


Buckle Up Montana Programs 


Covariates corresponding to BUMT activity were: 


1. An indicator variable taking on the value 0 if the corresponding survey site was not located in a county 
with an active BUMT coalition, and 1 if it was in a county with an active coalition 


2. A quantitative variable indicating the level of funding associated with that coalition in that year. 
Coalition budgets were obtained through coalition coordinators. The research team contacted coalition 
coordinators directly, asking for both budget and expenditures for all available fiscal years. Budget 
information was recorded for each coalition during each fiscal year available. To calculate per capita 
BUMT expenditures, we used 2010 U.S. Census data to determine the total number of residents in the 
(potentially multi-state) domain covered by each coalition. We then divided total expenditures in a given 
year by that population size. We did not attempt to use year-specific populations, since those data would 
have likely been unreliable. 


Selective Traffic Enforcement Program 


Covariates corresponding to STEP activity varied depending on what body received the funding. We 
needed to account for multiple funded groups operating in the same vicinity (for example, when both a 
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county and a municipality had separate STEP sites near one another). To do this, we constructed the 
following covariates at every spatial scale (counties, DOJ MHP Regions, municipalities): 


1. An integer-valued covariate corresponding to the number of STEP sites/activities present in that spatial 
region in a given year. 


2. A quantitative covariate corresponding to the total hours of additional enforcement in that spatial 
domain in that year. 


3. An integer-valued covariate corresponding to the number of days during which STEP activities 
occurred. 


We then divided each of the covariates above by the area of the jurisdiction (for example, the area of the 
county, municipality, or DOJ MHP region) and the population of the spatial domain. We summed these 
values for each site to generate three aggregated STEP intensity measures, one corresponding to the 
number of STEP locations nearby, one corresponding to total proximal STEP hours with which those 
sites were manned, and one corresponding to total proximal STEP days. For days labeled TBA, we 
assumed two days of STEP activities occurred. City Population covered by STEP program found using 
2010 Census data here: 


http://fact_nder2.census.gov/faces/tableservices/jsf/pages/productview.xhtml?src=bkmkhttps://doj.mt.gov 
/highwaypatrol/montana-highway-patrol-district-offices/ 
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2012 


Figure A1.1. Montana STEP allocations across the three years analyzed here. 


Media Campaign Data 
2011 


Data were provided through MDT (Pam Buckman). Media expenditures were aggregated at the city level, 
and assumed to have impacts only within the limits of the associated city. We used actual paid and bonus 
units wherever available. For Cable, only projected units were provided. Radio units for Bozeman and 
Belgrade were grouped, so we assigned the same values to both Bozeman and Belgrade. We had no 
information on billboard/outdoor advertising usage in 2011. 


2013 


For 2013, we pulled information from MDT_MemorialDay2013_Written_Form. Data were provided in 
aggregate across media venue. We divided total paid spots and total bonus spots by the number of 
markets covered to attain a metric for effort per market. Cable was not reported on a per-media-market 
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basis, and was thus excluded. We had no information for an online campaign in 2013. For any sites not 
included in media reports, we set media cost values to 0. 


Driver Education Data 


Boundary shapefiles for Montana secondary school districts were extract by KRM on 25 June 2013 from 
the following site: http://www.landsat.com/montana-free-gis-data.html. At the guidance of Fran Penner- 
Ray, GEMS reports (Montana Office of Public Instruction 2010-2012) were extracted from the following 
ftp site: http://gems.opi.mt.gov/ProgramsAndCourses/Pages/TrafficEdSummaryReport.aspx 


MT DOJ HP 


We used the map and form provided by the Montana Department of Justice at the following web portal: 
https://doj.mt.gov/highwaypatrol/montana-highway-patrol-district-offices/. Madison County split nearly 
equally into two parts, however we labeled it as being in District 7, since its population centers are 
predominantly located in District 7. City land areas were obtained via Wikipedia. 
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Model Covariance/Random Effects Structure 


Seat belt survey data were collected at particular sites, in particular counties, during particular years. We 
anticipate that extraneous factors likely differed between counties and sites, and across years of data 
collection, and want our modeling framework to account for that extraneous variation. Since we are only 
using data from three years at the preliminary modeling stage, we currently treat Year as a fixed effect; 
however, we account for variation at the site and county levels through incorporation of random effect 
terms for each of these factors. A given site always falls in the same county across all sampling events, 
thus we take Site to be nested in County. Below is a portion of the dataset that contains seat belt survey 
data, as well as the Site and County within which each data point was collected. 


Table A2.1. Example of NOPUS data structure. Source: NOPUS. 
Without restraints | With restraints 


Silver Bow 


Silver Bow 


Jefferson 


Jefferson 


Cascade 


Cascade 
Toole 
Toole 


Covariance Structure 


At the coarsest spatial level, we assume that factors like traffic density, regular patrolling patterns, and 
social and educational factors likely differ from county to county. Therefore, although we take seat belt 
compliance within a given county to be independent of seat belt compliance in all other counties, we 
expect survey data within the same county to be similar. Mathematically, we can formalize this 
assumption in the context of a covariance matrix. The assumption that observations in different counties 
are independent of one another translates to the block-diagonal correlation structure shown below. The 
four counties included are the ones in the data snippet above: Silver Bow, Jefferson, Cascade, and Toole. 


x Silver Bow 0 0 0 
0 x Jefferson 0 0 
Var(Y) = 
) 0 0 x Cascade 0 
0 0 0 x Toole 
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Within a particular county, correlations exist between sites. The within-county correlation structure 
(corresponding to & siyer Bow in the correlation structure above) for the Silver Bow county data snippet 
above can be written out such that year could also be treated as a random effect. 


Year could also be treated as a random effect, and is incorporated here as well. The dsijyer gowterm 
encapsulates the commonality between all observations in Butte-Silver Bow County. os;¢2 encapsulates 
the commonality between all surveys conducted at the same site. dyeq, captures commonalities between 
all surveys conducted in the same year. Since all of the surveys included below (Site I-01 and I-02 in 
2011 and 2012) occur in Silver Bow County, all terms in this matrix have a common baseline correlation 
of Osityer Bow- Observations at the same site have an additional term, 05;2 as do observations in the same 


year, Oyear. 
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Appendix 3. Examinations of Colinearity 
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Figure A3.1. Pairs plot for paid media by type (TV, radio, cable). 
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Figure A3.2. Pairs plot for district STEP effort. 
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Figure A3.3. Pairs plot for county STEP effort. 
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Figure A3.4. Pairs plot for city STEP effort. 
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Appendix 4. Terminology 


Binomial Variable 


A random variable describing the number of successes in a set of N trials. In the seat belt14survey data, 
we took a success to be an occupant wearing a seat belt, and the number of trials to be the total number of 
occupants observed. 


Covariate or Factor 


A descriptor thought to impact the response variable. In the occupant protection scenario, covariates 
thought to impact seat belt compliance rates include the drivers’ education completion rate in a given 
county, the weather, day of the week, and time of day under which data collection occurred, the presence 
and scale of different occupant protection programs operating near the survey site, etc. 


Fixed Effect 


A covariate whose effect is estimated explicitly for all levels included in the model. For example, in this 
case, we treat Stratum as a fixed effect, since we are explicitly interested in the impact of each level of 
stratum. 


Generalized Linear Model 


A model that is linear in the predictors (i.e., the composite effects of all covariates incorporated into the 
model can be expressed as a linear combination), but requires some transformation of the response 
variable. 


Logistic Regression 


A generalized linear model frequently used for modeling binomial data, which employs a logit link 
function, and results in estimates that link changes in covariate values to changes in the probability that a 
particular outcome occurs. In this case, we used logistic regression to estimate the impacts of the different 
occupant protection programs (the “covariates") on the binomial data collected during the seat belt 
surveys. 


Nested 


A situation in which a covariate is a subset of another covariate; the smaller set is said to be "nested" 
within the larger set. In this case, each site resides within a county (for example, [06 is in Silver Bow 
county), and therefore sites are nested within counties. 


Random Effect 


A factor covariate whose effect is explicitly estimated for every level included in the model, but which is 
incorporated into the model parametrically only through the variance across all covariate levels. In this 
case, we treat County as a random effect, since we are not explicitly interested in the effects of each 
county, but rather want to adjust the model for underlying differences that weren't incorporated into our 
covariate set. 
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